Add quant mode for qwen3.5 - #2096
Closed
Akshay Sonawane (apsonawane) wants to merge 5 commits into
Closed
Conversation
Copilot started reviewing on behalf of
Akshay Sonawane (apsonawane)
April 22, 2026 16:00
View session
Akshay Sonawane (apsonawane)
marked this pull request as draft
April 22, 2026 16:02
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new quant_mode option to the Qwen3.5 model builder to control mixed-precision quantization behavior (default/hybrid/int4), with corresponding validation in the shared extra-options parsing.
Changes:
- Added
quant_modevalidation incheck_extra_options(builder.py). - Implemented quantization strategy selection in the Qwen3.5 builder based on
extra_options["quant_mode"]orQWEN35_QUANT_MODE(qwen.py).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
src/python/py/models/builders/qwen.py |
Reads quant_mode and applies INT8 overrides for selected node subsets under different modes. |
src/python/py/models/builder.py |
Validates quant_mode values provided via --extra_options. |
Copilot started reviewing on behalf of
Akshay Sonawane (apsonawane)
April 23, 2026 01:44
View session
Member
|
#2078 is adding many unittest on modelbuilder, that would be great if you add for your changes once this PR is merged. |
1 similar comment
Member
|
#2078 is adding many unittest on modelbuilder, that would be great if you add for your changes once this PR is merged. |
Akshay Sonawane (apsonawane)
marked this pull request as ready for review
April 23, 2026 20:49
Contributor
Author
|
Created a new PR to use existing int4_algo_config instead of creating a new one: #2100 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new
quant_modeoption for configuring quantization behavior in Qwen model builders, allowing users to balance inference speed and model accuracy. The changes validate thequant_modeparameter, document its usage, and implement three quantization strategies: "default", "hybrid", and "int4".Key changes:
Quantization Mode Validation and Configuration
quant_modeoption in thecheck_extra_optionsfunction to ensure only supported values ("default", "hybrid", "int4") are accepted.quant_modefromextra_optionsor theQWEN35_QUANT_MODEenvironment variable, defaulting to "default" if unset.Quantization Strategy Implementation
"default": INT8 quantization for all linear attention and MLP layers (most accurate)."hybrid": INT8 for linear attention projections, INT4 for MLPs (balanced speed and accuracy)."int4": INT4 for all layers (fastest, may degrade quality).Documentation